Dynamic range is the ratio between intensities of the brightest and darkest recordable parts of an image or scene. A scene that ranges from bright sunlight to deep shadows is said to have a high dynamic range, while indoor scenes with less contrast will have a low dynamic range. Note that depending on the scene contrast, it may or may not be possible to capture the entire range with a digital camera. In recording scenes with very high dynamic range, digital cameras will make compromises that allow the capture of only the part of the scene that is most important. This compromise is needed because no camera or output device of any kind (including the human eye) can reproduce the nearly infinite dynamic range that exists in real life.
In an input image which contains a level of background noise, the noise level defines the darkest part of the image for the purpose of the dynamic range. Amplifying the brightness of the darkest parts of an image containing noise tends to decrease the dynamic range, as the darkest meaningful parts of the image, defined by the noise level, are increased in brightness.
Electronic cameras, based on CCD detector arrays, are capable of acquiring image data across a wide dynamic range on the order to 2500:1. This range is suitable for handling most illumination variations within scenes (such as high contrast scenes taken on a sunny day). Typically though, this dynamic range is lost when the image is digitised, compressed by a lossy compression algorithm (like JPEG or MPEG) or when the much narrower dynamic ranges of print and display media are encountered. For example, most images are digitised to 8-bit/colour band (256 grey levels/colour band) and most displays and print media are even more limited to a 50:1 dynamic range.
It would be desirable to try to find some transform which can compress the dynamic range of an image to allow more detail in the original image (which may be colour or non-colour, still or video) to be reproducible, while preferably preserving the natural look of the transformed image. Let us call such a transform an “image improvement transform”.
Since it is important to be able to improve images (video) in real time the computational efficiency is an important parameter of image improvement systems. To provide low power consumption, smallness in size and the low cost of an integrated circuit (IC) or other device implementing an image improvement algorithm, it would be desirable to provide an analogue implementation of an image improvement algorithm. Another advantage of an analogue implementation of image improvement algorithm is that would allow the avoidance of the addition of digitising noise caused by analogue to digital conversion (ADC).
It would also be desirable to provide an improvement to lossy data compression algorithms such as JPEG, to allow a greater amount of manipulation of compressed/decompressed images.
Let us consider the current popular image enhancement algorithms.
Homomorphic filtering is an image enhancement technique which uses the following algorithm:Iout=exp[HPFΩ(log(I))]=exp[log(I)−LPFΩ(log(I))],
where HPF is a high-pass 2D spatial filter function.
U.S. Pat. No. 5,991,456 describes an image enhancement algorithm system named Retinex™. This uses the following algorithm:
            I      out        =                  ∑                  i          =          0                N            ⁢                        W          i                ⁡                  (                                    log              ⁡                              (                I                )                                      -                          log              ⁡                              (                                                      LPF                                          Ω                      i                                                        ⁡                                      (                    I                    )                                                  )                                              )                      ,
where Wi are constants.
A disadvantage of this algorithm is that it is incapable of optimisation of the high-frequency (spatial) components
  (      with    ⁢                  ⁢    frequencies    ⁢                  ⁢    higher    ⁢                  ⁢    than    ⁢                  ⁢                  max                              i            =            1                    ,          …          ⁢                                          ,          N                    ⁢              (                  Ω          i                )              )of an image, thus sharp transitions between dark and bright regions of an image will exist on the transformed image. Thus a disadvantage of the Retinex algorithm is the excessive contrast of small high-contrast details of images. A further disadvantage is its computing power requirements. Typically, image processing takes hours on a conventional computer workstation. Therefore, Retinex is ill-suited to most practical applications
Techniques known as histogram modification (or equalisation) algorithms are also known. There are many histogram modification algorithms. Such algorithms modify the transfer function H(..) of the whole image:Iout(x, y)=H(I(x, y)).
More particularly, histogram equalisation (cumulative histogram-based) as well as other histogram modification-based algorithms may be described using the following equation:
                              I          out                =                ⁢                              1            M                    ⁢                      CumHistogram            ⁡                          (              I              )                                                              =                ⁢                              ∑                          i              =              0                        N                    ⁢                                    θ              ⁡                              (                                  I                  -                                      I                    i                                                  )                                      ⁢                          〈                              δ                ⁡                                  (                                      I                    -                                          I                      i                                                        )                                            〉                                                                        ≡                    ⁢                                    ∑                              i                =                0                            N                        ⁢                          θ              ⁢                              (                                  I                  -                                      I                    i                                                  )                            ⁢                              1                M                            ⁢                                                ∑                                      k                    ,                    l                                                  ⁢                                  δ                  ⁡                                      (                                                                  I                        ⁡                                                  (                                                      k                            ,                            l                                                    )                                                                    -                                                                        I                          i                                                ⁡                                                  (                                                      k                            ,                            l                                                    )                                                                                      )                                                                                      ,            
where M is the number of pixels in the image I, Ii are the levels of grey (e.g. in case of grayscale images N=255, I0=0, I1=1, . . . , I255=255),
            δ      ⁡              (        x        )              =                            1                                                    if              ⁢                                                          ⁢              x                        =            0                                                0                                                    if              ⁢                                                          ⁢              x                        ≠            0                                ,            θ      ⁡              (        x        )              =                            1                                                    if              ⁢                                                          ⁢              x                        ≥            0                                                0                                                    if              ⁢                                                          ⁢              x                        <            0                                ,and (k,l) are the pixel coordinates. The averaging operation .. could be considered as the low-pass filtering LPFΩ with the cut-off frequency
      Ω    =    0    ,            so      ⁢                          ⁢              I        out              =                  ∑                  i          =          0                N            ⁢                        θ          ⁡                      (                          I              -                              I                i                                      )                          ⁢                                            LPF              0                        ⁡                          (                              δ                ⁡                                  (                                      I                    -                                          I                      i                                                        )                                            )                                .                    
Histogram equalisation algorithms can be limited in utility in cases when the histogram of the whole image is uniform, whereas some parts of the image are too bright and the others are too dark. In such cases different transfer functions should be applied to different regions of images.
Local histogram equalising (also referred to as adaptive histogram equalising) techniques are also known. The techniques are adaptive in the sense that in different regions of a picture different transforms are applied to equalise the histogram of each particular region. There are many implementations, which can be represented by the formula:
            I      out        =                  ∑                  i          =          0                N            ⁢                        θ          ⁡                      (                          I              -                              I                i                                      )                          ⁢                              LPF            Ω                    ⁡                      (                          δ              ⁡                              (                                  I                  -                                      I                    i                                                  )                                      )                                ,
where Ω is the spatial frequency related to the size of regions with equalised histogram (if the size of “window” is r, the size of an image is L, than
            Ω      =              L        r              )    .
However, known local histogram equalisation (adaptive histogram equalisation) techniques are highly computationally intensive and difficult to embody in real time applications. Furthermore, in many cases a rectangular window around each pixel is used for histogram equalisation in this window, thus such transforms cannot be considered as spatially smooth. Such non-smoothness of a transform could increase the noise and artefacts of transformed images.
It is an object of one or more aspects of the present invention to provide a method of improving an image. Preferably, the image may be created with digital or analogue data. Preferably, the method should be suitable for application to both colour and non-colour images.
Another object of one or more aspects of the present invention is to provide a method of improving images in terms of dynamic range compression, colour independence from the spectral distribution of the scene illuminant, and colour/lightness rendition.
Another object of one or more aspects of the invention is to provide analogue implementations of the suggested image improvement algorithm. Such implementations will generally be small sizes and have low power consumption, which are desirable characteristics in many image capture devices.
Still another object of one or more aspects of the invention is to improve lossy image compression procedures, such that after compression-decompression images can retain a relatively wide dynamic range, which is important if certain types of post-processing are to be performed, for example in medical image processing techniques (such as those used in X-ray and Magnetic Resonance Imaging (MRI scans) or in the manipulation of photographic images using image editing software.
In accordance with one aspect of the present invention, there is provided an image processing method comprising the step of processing an input signal to generate an adjusted output signal, wherein the intensity values I(x,y) for different positions (x,y) of an image are adjusted to generate an adjusted intensity value I′(x,y) in accordance with:Iout=Σi=0Nαi(I)LPFΩi[Pi(F(I))]·Qi(F(I))·Qi(F(I))+β(I),
where Pi(γ) is an orthogonal basis of functions of γ defined in the range 0<γ<1; and Qi(..) are antiderivatives of Pi(..): Qi(F(I))=∫0F(I)Pi(η)dη or an approximation thereto, LPFΩ[..] is an operator of low-pass spatial filtering; a Ωi is a cut-off frequency of the low-pass filter, and F(..) is a weighting function.
The family of possible transforms, which could be represented by the above algorithm, are referred to herein as the Orthogonal Retino-Morphic Image Transform (ORMIT).
The functions Qi(..) may be approximations to the antiderivatives of Pi(..) within a range of 25%, more preferably 10%, of variation from the antiderivatives. Ω is preferably non-zero. F(..) preferably varies with I in an asymmetrical manner, β(I) is preferably not equal to zero. Ωi is preferably different for different i such that the transformation has a different degree of spatial nonuniformity in different brightness domains.
The image improvement algorithm is preferably asymmetric in the brightness domain, such that the algorithm has a greater effect on dark regions of an image than relatively bright regions. The algorithm preferably performs dynamic range compression so as to displace relatively dark portions of the image within the range of intensity values to a greater extent than relatively bright portions, thereby achieving this asymmetry. The image contrast in the dark regions is preferably also increased to a greater extent than that in the light regions. The effect of this is to mimic the behaviour of the human eye, which has a relatively high dynamic range extending well into the dark regions of an original image, by reproducing these dark regions in a lighter region of the reproduced image. Furthermore, when used, reversibly, in combination with a standard lossy compression technique, as will be described in further detail below, the dynamic range compression functions to increase the amount of useful data retained in the dark regions of an image. Since the human eye has a very high dynamic range (apprx. 105:1) whereas the dynamic range of the optic nerve is about 100:1, it is clear that the eye performs a transform which strongly compresses the dynamic range.
The algorithm according to this aspect of the invention preferably includes an element of spatial nonuniformity in the transfer function which is applied to the image. That is to say, the transfer function may vary across the image, as in the case of local histogram equalisation.
Furthermore, the shape of the transfer function applied is preferably nonlinearly adaptive in the spatial domain. That is to say, the transfer functions which are applied in different parts of the image need bear no relation to one another in shape, to improve the handling of contrast present in an original image. The transform preferably makes use of orthogonal functions, thereby to improve computational efficiency. The orthogonal functions are in one embodiment Legendre polynomials, in another piecewise linear mapping functions. These transforms are preferably carried out in the analogue domain.
The algorithm is preferably smooth both in the brightness domain and the spatial domain.
A further preferred characteristic of the algorithm is reversibility.
According to a further aspect of the invention there is provided a method of compressing an image signal, comprising conducting a reversible dynamic range compression algorithm to transform the image signal to an enhanced image signal, and conducting a lossy compression algorithm to compress the enhanced image signal to generate compressed image data.
The method preferably comprises generating parameters relating to the dynamic range compression algorithm, and storing same in combination with the compressed image data.
According to a yet further aspect of the invention there is provided a method of decompressing compressed image data, comprising conducting a decompression corresponding to a lossy compression algorithm, and conducting a reverse dynamic range compression algorithm to provide a decompressed image signal.
In one embodiment, a dynamic range compression algorithm is used reversibly in combination with a standard lossy compression technique (such as JPEG), with parameters describing the image improvement transform accompanying the compressed image data to enable the reverse transform when decompressing the image. In this sense, the algorithm is to be considered “reversible” if the amount of data describing the reverse transform is not significantly greater than the amount of standard image compression data. Preferably, the amount of transform data is significantly less than, i.e. less than half the amount of, the standard image compression data. In one embodiment, the transform data is inserted in a header part of the compressed image file.
In a preferred embodiment, the intensity value I(x,y) for each position (x,y) of a greyscale image is adjusted to generate an intensity value Iout(x,y) (improved image) for each position in accordance withIout=α·Σi=0NLPFΩ[Pi(F(I))]·Qi(F(I))+(1−α)I
Here α is the “strength” of transform, which is preferably, in the case of image enhancement between 0 and 1. Typically, α may take a value in the region of 0.5. The nonuniformity parameters Ωi may take different values for different i to achieve best enhancement while retaining natural appearance. It may be convenient to set all to a constant value Ω as above.
The proper selection of α (which is the strength of transform), spatial nonuniformity of the transform (the “window” of the transform) represented by Ω, the weighting function F(..) and basic functions Pi(..) provides good quality image improvement, e.g. dynamic range compression.
The image is initially represented by electronic format (either in analogue or in digital form). The proposed algorithm works both with colour and with grayscale images. In case of colour images the proposed algorithm could modify either the intensity of image (whereas the hue and saturation are unchanged) or individual colour channels of an image (thus providing the colour correction). Different approaches to colour-correction of images are also suggested.
The procedure of the reversed transform is suggested. This procedure allows restoring of the original image from the transformed one.
A novel procedure for image compression is suggested. The procedure consists of an image transform, compressing the dynamic range of an image, as the first stage and a standard lossy image compression algorithm, such as JPEG or JPEG2000, as the second stage. After decompression and reversed ORMIT transform the quality of dark regions of an image is much better than the quality of such regions after JPEG compression-decompression (with the same size of compressed file). Thus the dynamic range of images, compressed by the suggested procedure, is increased.
Schematics, implementing different approaches to analogue hardware implementation of the image improvement system, are suggested. The first schematic is based on a polynomial approach, whereas the second is based on a piecewise linear mapping function approach. Also suggested is a simplified version of ORMIT, wherein the spatial nonuniformity parameter Ω is set to zero, which has a simpler implementation in analogue circuitry.
In this case, the ORMIT formula reduces to:
            I      ′        =                  ∑                  i          =          0                ∞            ⁢                        〈                                    P              i                        ⁡                          (              I              )                                〉                ⁢                              ∫            0            I                    ⁢                                                    P                i                            ⁡                              (                η                )                                      ⁢                                                  ⁢                          ⅆ              η                                            ,where I(t) is the input (video) image signal, I′(t) is the output signal, and where Pi(x) defines a basis of orthogonal functions and the operator .. means the arithmetic mean (or mathematical expectation) value for the image. Versions are well suited to the enhancement of video images in real time.
A motivation for performing image enhancement in analogue circuitry immediately after capturing a video signal from an image sensor (e.g. CCD) is the fact that it provides potentially the widest dynamic range of the output image of the whole image processing system (e.g. the video capture system, image transmission or recording system and the video output system). It results in the visibility of details of an image which would otherwise have been below the level of noise caused by the transmission, recording and reproduction (in the case of analogue video cameras), or below the noise of digitizing by an analogue-digital converter (ADC) and lossy compression (like MPEG).
For real-time applications, the image of a subsequent video-frame is preferably processed on the basis of information from a previous frame, preferably the immediately preceding frame. This is justified because statistically video images change predominantly just slightly from frame to frame.
Preferably, the enhancer functions with an asynchronous regime, to allow the device to be easily installed both into digital and analogue cameras.
The correcting transform of the video signal, performed by the enhancer, is preferably smooth and therefore reduces the effect of amplification of visible noise, which is one of the drawbacks of standard histogram equalising technique (see William K. Pratt, “Digital Image Processing”, John Wiley and Sons, 1978, pp. 307-318).
Advantages of an analogue device based on the proposed technique include smallness in size and significantly less power consumption in comparison with a digital implementation.
The use of analogue technology for nonlinear image processing allows maximizing of the dynamic range, since the nonlinear processing takes place before the digitizing.
The described analogue systems are capable of real-time video processing.
The suggested algorithm can be implemented as a monolithic analogue Integrated Circuit (IC); such a chip could be installed both into digital and into analogue video cameras. The size of an analogue IC implementing the histogram equalization could be relatively small and the power consumption relatively low in comparison with digital systems. Such an IC could be used e.g. in security cameras, where the cost and the power consumption could be critical. Other applications are in high-speed cameras, where digital image processing systems might not be fast enough, or in highest quality cameras, where the wide dynamic range of CCD (which could be higher than 2500:1) should be maximally preserved. In this case the use of the analogue equalizers could be justified, since it will allow avoiding the noise of digitizing, caused by the ADC.